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' . ABSTRACT i * / 

An examinee, is required to express his confidence in the correct- 
ness of each choice of a jauXtiple-choic^e item in a probabilistic test. 
. For the responses to be valid' indicators the confidence expressed in 
^ each choice should be determined by .an examinees ^\knoWledge« This study 
assessed the relationship of the certainty of examinees' responses to 
knowledge and selected personality traits. It wds found that a reliable 
certainty of response was exhibited by examinee's. This 'certainty measure 
had a ^derately high (.62) relationship to examinees knowledge. The 
certainty of response was* also related to risk taking holding knowledge' 
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constant . ' ' ' , < 
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THE CONTRIBUTION OF* SELECTED PERSONALITY TRAITS AND KNOWLEDGIe 
TO RESPONSE BEHAVIOR ON A PROBABILISTIC TEST , • 
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Objectives ^ ' ' . ^ 

The multiple-choice type test can be administered using a response^ 
system which permits an examinee to indicate the subjective probability 
of the correctness of' each of thi^ chpices to an item* Such a test is 
referred to as^^ probabilistic test* This study was conducted to elaborate 
on the inforjnation which has been previously reported on the response be- 
havior of *Ss on probabilistic tests. If the results are found to be similar 
to previous studies, then this study would assist in the generalization of 
probabilistic test characteristics. If the findings were found to be <lif- 
t^x^it^^^^exi this study would assist in delimiting previous findings. 

The research hypotheses geneuf^^ti for this study were that (a) a reliable 
variable of certainty of respon^e'^an be* measured using a probabilistic test, 
(b) this certainty of response can be rellablly measurW with knowledge held 
constant, and (c) this certainty of response measure Is significantly related 
to selected personality traits holc/ing knowledge constant. 

t heoretical Framework 

dePinetti, (1965) has considered the theoretical implications of a 
number of scoring systems which attempt to reflect partial knowledge by 
using subjective probabilities in probabilistic testing. Rlppey (1970) 



conducted a comparative study of different probabilistic scoring functions. 
In 'both studies evidence was presented that simple scoring functions hai 
relatively positive characteristics when compared to complex scoring systems.' 
By using relatively sample scoring functions, seVQSjJL studies have found that 
the reliability of a test can be Increased. 

' If Ss respond to a probabilistic test in the Intended mannerv their 
responses should be determined primarily by the knowledge which is claimed 
to be measured by the probabilistic . test . If the responses of Ss are par- 
tially governed instead by personality traita, then the validity of the 
test may be altered. Hansen (1971) investigated the Influence of variables 
other than knowledge on the responses of Ss under probabilistic cqpdltlons. 
He found that Ss do respond to test items under probabills conditions . 
with a certainty characteristic that was not accounted for by their knowledge 

This study yas conducted to see if similar results to those in the 
Hansen study would be found using a different probabilistic scoring function, 
different testing conditlo^j^ and only a partial overlap of personality 
measures. 

4 

Methods 

A simple scoring function was selected for this study in which Ss 
expressed their degree of confidence in each choice by distributing 5 points 
across the five choices on the test Item^j each point represented a proba- 
bility of 0.2. The Ss had only 5 points to distribute across the choices 
for each item. The item score was the number of points S placed on the cor- 
rect answer. ' , 

In the Hansen study, the spherical scoring function was used. That 
function differed from the function In this study In the options available' 



to the S3 as well as the method of , scoring an item. 

Studies of probabilistic tests have been reported which i^easure achieve- 
ment in a course. Hansen^s study was an example. In contrast, a vocabulary 
test was ^elected for this stiidy which was not related to the »course objectives. ^ 
It was thought tliere was a lack of information regarding the characteristics^ ot 
probabilistic tests which measure non-course type objectives. The vocabulary 
test consisted of , 24 items from the I.E.R. Intelligence Scale (1946). Items 

-.J - 

were randomly selected from each of the fiv^ levels of the intelligence scale. , 

Personality measures of external control, risk taking* and cautiousness 
were obtained to assess personality traits that might bias the vocabulary 
scores under the testing condition of expressing subjective probabilities » 
The internal-external sc^le identified individuals according to differences 
in a belief in external Cijntrol (Rotter, 1966). The Kpgan and Wallach (1964) 
questionnaire assessed an individual *s^risk taking behavior and the Gordon^ (1956) 
Personal Inventory measured the general trait of cautiousness. Only the risk 
taking measure was among the instrtmients used in the Hansen study » 

Hansen developed an index of certainty from the responses of Ss on a 
probabilistic test. This index of certainty is the average of the absolute 
deviations from a l-l-'-JL-l-l response < (assuming 5-choice item). Before com- 
puting the index all responses are converted to probabilities. The index is 
at a maximum when a single choice is assigned a probability of 1.00.. This 
would be indicated by 3 placing 5 points on a single choice. The inde;c is 
at a minimum when a probability of 0.20 is assigned to each choite as indi- 
cated by a 1-1-1-1-1 response. ' * 

The level of certainty of an individual might be accounted for by the 
knowledge of S. Therefore, the certainty index was adjusted for the linear 



relat;Lon8hlp of certainty an^ knowledge. -A simple regression equation was 
developed uuing the certainty index as the criterion variable and the vocabu- 
lary knowledge score as the predictor variable. Following the procedure of 
Hansen^ residual SLbejtween observed certainty scores and" predicted, certainty 
scores were calculated as measures of certainty with knowledge held^onstant • 
B'stimates ofSr^laljilltie^f or these indices w^re calculated using analysis 
of variance • 

* K. 4 
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Data Source 

The Ss used in this study were graduate students enrolled in educational 

measurement courses in the School of Education at Indiana University. /These 

courses were reqyired for several degree programs and had students enrolled 

from several specialization areas. A sample of 56 Ss consisting of 40 females 

and 16 males was selected. The mean age of the sample was 26.3 years. 

Prior to taking the vocabulary test all Ss w^re given a brief training 

# » 

session. The first part was a presentation by Es of the scoring function. 
The second part allowed Ss to use the probabilistic scoring system on practice 
test Items.. 

Results ^ ^ ' 

In Tahiti* merna» standard deviations » and reliability estimates ate 
reported for the krfbwiedge of vocabulaty score» the observed certainty score» 
and the residual certainty' score. 

^ INSERT TABLE. X ^ . 

The knowledge of vocabulary test scores had a consistency reliability 
of .66. The' observed certainty meaauf^ h^d a reliability of .82 and the 
residual certainty measure had a reliability of .32. 



The correlation between the knowledge of vocabulary scores and the * 
observed certainty scores was moderately high, .^2. 

The correlation coef f4.cients between the personality trait scores and 
the knowledge of-vocabulary, the observed certainty and -the -resldiial aertainty_ 
i^easures are reported in Table 2. / ' ^ 
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The 'risk taking scopes correlated significantly with both the observed 
and residual certainty measures (p<.05). All other cprrelation coefficients 
were insignificant* > 

Importarce^of th^ Study 

The Ss tended td respond to the vocabulary iteftis with a behavior that 
was consistently either certain or uncertain* ^^This behavior was moderately 

■ ■ ' . ' y 

high In relationship to knowledge of vocabulary but %as not totally accounted 

for by their knowledge. The low but significant correlation between risk 
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taking and residual certainty indicated that the response behavior was partly^ 
a function of Ss preference for risky optlon6. Those iS.s who scored higl^er in 
risk taking tended to be more certain in their responses than was typical for 
Ss with the same knowledge score* _ , 

In comparison to the Hansen study, both studies found reliable measures 
of observed certainty and residual certainty* The reliability of observed | 
certainty wad similar for the two studies but the reliability of resjidual cer* 
tainty was appreciably lower in this study than in the Hansen study. Both 
j^tudles found that a risk taking measure accounted for certainty of ^response 
behavior holding knowledge constant. A much higher relationship was found in 
this study between knowledge and certainty of^ response which contributed to 
the lower residual certainty reliability* One possible explanation of this 



difference Is that under course examinations of the Hansen study, additional 
factor's entered into the' response behavior of Ss that did not with t)ie*vocab- 
ulary, test. AlthouglT'th^e is Increased evidence that factors other than 

into the response behavior on probabf listlc tests, there is 
ho evidence in .either, study that these factors a^a- igore operative thaiTlin^ 
traditional tests. 




TABLE 1. MEANS, STANDARD DEVIATIONS, AND' RELIABILITY COEFFICIENTS FOR 
MEASURES OF KNOWLEDGE OF VOCABULARY, OBSERVED CERTAINTY AND RESIDUAL 
CERTAINTY (N=«56). 



MEASURES 



"MEANS 



STANDARD RELIABILITY 
DEVIATIONS COEFFICIENTS 



Knowledge of Vocabulary 
Observed Certainty 
Residual Certai,nty 



50.57 
.54 
.00 



12.32 
.17 
. -.13 



.66 . 

.82 

.32^ 



'-Estimated using the reliability of a difference score. 



TABLE 2. CORRELATION COEFFICIENTS OF KNOWLEDGE OF VOCABULARY, OBSERVED 
CERTAINTY,' AND RESIDUAt's CERTAINTY WITH PERSONALITY TRAIT SCORES 
(N-56). , 



' KNOWLEDGE OF OBSERVED- RESIDUAL 
PERSONALITY TRAIT VOCABULARY CERTAINTY CERT^NTY 



External, Control " -.131 -.008 .095 

Risk Taking •■ .038 -.283* -.328* 

Cautiousness .083 -.011 -.081 



V.05 
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